Reliability testing of a case-leveling framework for assigning level of difficulty of pharmacist's initial patient medication assessments.

OBJECTIVE To develop and test the reliability of a case-leveling framework for assigning level of difficulty of the pharmacist's task for initial medication assessments conducted by pharmacists integrated into family physician offices. DESIGN Descriptive study. SETTING Seven family practice sites in Ontario from June 2004 to July 2006. PATIENTS Patients referred by their family physician for pharmacist assessment. INTERVENTION Individual medication assessments, monitoring, and follow-up by pharmacists. A case-leveling framework was developed with three levels of complexity (graded as I, II, or III) including specific descriptors and practice-based examples. Reliability was assessed between two standardized assessors and between one assessor and project pharmacists. Project pharmacist feedback was elicited through an e-mail survey. Reliability is reported using the kappa statistic. MAIN OUTCOME MEASURES Reliability of a case-leveling framework and helpfulness of the framework as reported by pharmacists. RESULTS 53 patient cases were evaluated for interrater reliability between standardized assessors. The mean (+/- SD) case level assigned was 1.8 +/- 0.68, and the kappa was 0.62 (95% CI 0.44-0.79), indicating a substantial strength of agreement between raters. For the second reliability test, 52 cases were rated, with a level of agreement between project pharmacists and the external assessor of 0.46 (95% CI 0.27-0.65), indicating moderate agreement. Feedback resulted in slight revisions to the original framework. CONCLUSION The case-leveling framework was a reliable method and can be used to determine the level of difficulty of patient cases in primary care.